Facial Expression Grounded Conversational Dialogue Generation

We present a novel conversational language model that is grounded with information about facial expressions. To our knowledge this is the first in-depth examination of grounding natural language models with facial cues. We train a neural language model that uses automatically detected facial action unit intensity information in images alongside text to generate conversational dialogue. We evaluate our model on a large and very challenging unconstrained real-world dataset from social media (Twitter), featuring 450,000 conversations with associated facial expressions. Systematic linguistic and crowdsourced analyses reveal the properties of our models: The facial expression grounding strengthens the sentiment of the resulting dialogue such that it is consistent with the valence of the facial expressions. Furthermore, the automatically generated conversational responses are rated as equivalent to the human gold-standard responses on relevance and emotion dimensions.